地震波的频域模拟在地震反演中起着重要作用，但在大型模型中仍然具有挑战性。作为有效的深度学习方法，最近提出的物理知识的神经网络（PINN）在解决广泛的偏微分方程（PDES）方面取得了成功的应用，并且在这方面仍然有改进的余地。例如，当PDE系数不平滑并描述结构复合介质时，PINN可能导致溶液不准确。在本文中，我们通过使用PINN而不是波方程来求解频域中的声学和Visco声学散射的场波方程，以消除源奇异性。我们首先说明，当在损失函数中未实现边界条件时，非平滑速度模型导致波场不准确。然后，我们在PINN的损耗函数中添加了完美匹配的层（PML）条件，并设计了二次神经网络，以克服PINN中非平滑模型的有害影响。我们表明，PML和二次神经元改善了结果和衰减，并讨论了这种改进的原因。我们还说明，在波场模拟中训练的网络可用于预先训练PDE-Coeff及时改变后另一个波场模拟的神经网络，并相应地提高收敛速度。当两次连续迭代或两个连续的实验之间的模型扰动时，这种预训练策略应在迭代全波形反转（FWI）和时置目标成像中找到应用。

具有更多数据，计算和参数的缩放语言模型在自然语言处理方面取得了重大进展。例如，由于缩放，GPT-3能够在内心学习任务上实现强烈结果。但是，培训这些大密度模型需要大量的计算资源。在本文中，我们提出并开发了名为Glam（通用语言模型）的语言模型系列，它使用稀疏激活的专家架构来规模模型容量，同时与致密变体相比，也产生显着更少的训练成本。最大的Glam具有1.2万亿参数，比GPT-3大约为7倍。它仅消耗了用于训练GPT-3的1/3的能量，并且需要一半的计算拖鞋进行推理，同时仍然在29个NLP任务中实现更好的整体零射击和一次性性能。

手术动作三胞胎识别提供了对手术场景的更好理解。这项任务具有很高的相关性，因为它为外科医生提供了背景感知的支持和安全性。当前改善绩效的首选策略是开发新的网络机制。但是，当前最新技术的性能大大低于其他手术任务。为什么会发生这种情况？这是我们在这项工作中解决的问题。我们提出了第一项研究，以了解现有的深度学习模型通过稳健性和解释的镜头的失败。首先，我们通过对抗优化方案研究了当前的现有模型。然后，我们通过基于功能的解释提供故障模式。我们的研究对提高性能和提高可靠性的关键是核心和虚假属性。我们的工作为外科科学中更具可信赖性和可靠性的深度学习模型打开了大门。

联合学习是分布式机器学习领域中的一个新兴概念。这个概念使甘斯能够从保留隐私的同时从丰富的分布式培训数据中受益。但是，在非IID设置中，当前的联合GAN体系结构是不稳定的，努力学习独特的功能并容易崩溃。在本文中，我们提出了一种新型的体系结构多流体，以解决非IID数据集的低质量图像，模式崩溃和不稳定性的问题。我们的结果表明，与基线Flgan相比，多流通量是平均20多个客户的稳定且性能的四倍。

多芯片芯片模块（MCM），而票面上提供性能和能效的单片大芯片减少了机器学习（ML）加速器的设计和制造成本。然而，统计MCM的ML编译器需要最佳，有效地解决复杂的优化问题，以实现这种高性能。其中一个问题是多芯片分割问题，在编译器确定在小芯片的MCM张计算图形操作的最佳分配和安置。作为搜索空间可用芯片的数目和节点的神经网络在数量呈指数级增长分区ML图形的多芯片模块是特别难。此外，由底层硬件施加的约束产生了一个有效解决方案非常稀疏的搜索空间。在本文中，我们提出使用深强化学习（RL）框架来发出可能无效分区候选人，然后由约束求解修正的策略。使用约束求解器可确保RL遇到稀疏空间中的有效解决方案，其经常足以与未经学习的策略相比较少的样本收敛。我们为策略网络制作的架构选择允许我们拓展不同的ML图形。我们的生产规模的模型，BERT，在真实的硬件的评估表明，使用RL政策所产生的分区达到6.11％和5.85％，比吞吐量随机搜索和模拟退火更高。此外，微调预训练RL政策减少了3小时至只有9分钟的搜索时间，同时实现了相同的吞吐量从头训练RL政策。

深度尖峰神经网络（SNNS）目前由于离散二进制激活和复杂的空间 - 时间动态而导致的基于梯度的方法的优化困难。考虑到Reset的巨大成功在深度学习中，将深入了解剩余学习，这将是自然的。以前的尖峰Reset模仿ANN的标准残留块，并简单地用尖刺神经元取代relu激活层，这遭受了劣化问题，并且很难实施剩余学习。在本文中，我们提出了尖峰元素 - 明智（SEW）RESET，以实现深部SNNS的剩余学习。我们证明SEW RESET可以轻松实现身份映射并克服Spiking Reset的消失/爆炸梯度问题。我们在Imagenet，DVS手势和CIFAR10-DVS数据集中评估我们的SEF RESET，并显示SEW RESNET以准确性和时间步长，最先进的直接训练的SNN。此外，SEW Reset通过简单地添加更多层来实现更高的性能，提供一种培训深舒头的简单方法。为了我们的最佳知识，这是第一次直接训练具有100多层以上的深度SNN。我们的代码可在https://github.com/fangwei123456/spike-element-wore-resnet上获得。

Transfer learning, where a model is first pre-trained on a data-rich task before being finetuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.

A recent study has shown a phenomenon called neural collapse in that the within-class means of features and the classifier weight vectors converge to the vertices of a simplex equiangular tight frame at the terminal phase of training for classification. In this paper, we explore the corresponding structures of the last-layer feature centers and classifiers in semantic segmentation. Based on our empirical and theoretical analysis, we point out that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes, which breaks the equiangular and maximally separated structure of neural collapse for both feature centers and classifiers. However, such a symmetric structure is beneficial to discrimination for the minor classes. To preserve these advantages, we introduce a regularizer on feature centers to encourage the network to learn features closer to the appealing structure in imbalanced semantic segmentation. Experimental results show that our method can bring significant improvements on both 2D and 3D semantic segmentation benchmarks. Moreover, our method ranks 1st and sets a new record (+6.8% mIoU) on the ScanNet200 test leaderboard. Code will be available at https://github.com/dvlab-research/Imbalanced-Learning.

Weakly-supervised object localization aims to indicate the category as well as the scope of an object in an image given only the image-level labels. Most of the existing works are based on Class Activation Mapping (CAM) and endeavor to enlarge the discriminative area inside the activation map to perceive the whole object, yet ignore the co-occurrence confounder of the object and context (e.g., fish and water), which makes the model inspection hard to distinguish object boundaries. Besides, the use of CAM also brings a dilemma problem that the classification and localization always suffer from a performance gap and can not reach their highest accuracy simultaneously. In this paper, we propose a casual knowledge distillation method, dubbed KD-CI-CAM, to address these two under-explored issues in one go. More specifically, we tackle the co-occurrence context confounder problem via causal intervention (CI), which explores the causalities among image features, contexts, and categories to eliminate the biased object-context entanglement in the class activation maps. Based on the de-biased object feature, we additionally propose a multi-teacher causal distillation framework to balance the absorption of classification knowledge and localization knowledge during model training. Extensive experiments on several benchmarks demonstrate the effectiveness of KD-CI-CAM in learning clear object boundaries from confounding contexts and addressing the dilemma problem between classification and localization performance.

Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving. Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task. To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. The proposed PPGeo is performed in two stages to support effective self-supervised training. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only. As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks. Extensive experiments covering a wide span of challenging scenarios have demonstrated the superiority of our proposed approach, where improvements range from 2% to even over 100% with very limited data. Code and models will be available at https://github.com/OpenDriveLab/PPGeo.